We wrote this book to provide a framework for discussing the inevitable marriage of two ubiquitous concepts: machine learning and security. While there is some literature on the intersection of these subjects (and multiple conference workshops: CCS's AISec, AAAI's AICS, and NIPS's Machine Deception), most of the existing work is academic or theoretical. In particular, we did not find a guide that provides concrete, worked examples with code that can educate security practitioners about data science and help machine learning practitioners think about modern security problems effectively. In examining a broad range of topics in the security space, we provide examples of how machine learning can be applied to augment or replace rule-based or heuristic solutions to problems like intrusion detection, malware classification, or network analysis. In addition to exploring the core machine learning algorithms and techniques, we focus on the challenges of building maintainable, reliable, and scalable data mining systems in the security space.

Pedestrians walk under a surveillance camera, which is part of a facial recognition technology test in Berlin, Germany. Machine learning shines at tasks like this because it can recognize patterns and predict threats in massive data sets, all at machine speed. There is a temptation to hail major advances in technology as cure-alls for the challenges facing organizations and society today. The fanfare usually ends in disappointment, as the latest superhero technology doesn't live up to its expectations. Not surprisingly, machine learning, a domain within the broader field of artificial intelligence, has been hailed as the current be-all end-all answer in cybersecurity. As a result, it is currently at the peak of inflated expectations in Gartner's most recent Hype Cycle for Emerging Technologies.

The cyber threat landscape today forces organizations to constantly track and correlate millions of external and internal data points across a number of endpoints. It simply is not feasible to manage this volume of information on an ongoing basis with a team of people. Machine learning shines here because it can recognize patterns and predict threats in massive data sets, all at machine speed. By automating the analysis, cyber teams can rapidly detect threats and isolate situations that need deeper human analysis. While machine learning offers tremendous promise for cyber security, it has its share of shortcomings that need to be acknowledged in order to use it appropriately. Machine learning is not a panacea for increasing cyber resilience.

Artificial intelligence and machine learning are playing an increasing role in most aspects of enterprise technology. Because the challenges of cyber security are changing AI is playing an increasing role in defending enterprises from criminals. Malware is today evolving so fast that manual defences simply cannot keep up. Researchers at G DATA have been counting new malware types for many years. In 2007 they found 133,253 new malware specimens.